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(54) Adjustment layers for composited image manipulation 



(57) A method and system for compositing graphical 
images, wherein an advanced adjustment layer may be 
applied during a compositing process to a set of image 
layers 1 ... a ortoany subordinate subset of such image 
layers. One or more adjustment layers are applied to an 
intennediate merged image, generated by compositing 
previous image layers, and the adjusted result is stored 
as a temporary image. The temporary image is then in 



turn composited with the intermediate merged image. 
Any remaining image layers are then composited in with 
the intermediate merged image to generate a final 
merged image. The invention allows a user to apply a 
vast array of effects without requiring significant new 
knowledge on the part of the user. For example, if there 
are "A" adjustments and "T" transfer modes, the present 
invention allows AxT effects which leverage existing 
knowledge of the user of only A+T functions. 
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Description 

BACKGROUND OF THE INVENTION 

5 1 . Field of the Invention 

This invention relates to graphical image manipulation systems, and more particularly to a method and system for 
compositing graph teal images. 

'0 2. Description of Related Art 

A number of graphic image manipulation computer programs build up a final image by compositing several image 
layers together. An example of one such program is Adobe Photoshop 3.0, from Adobe Systems, Inc. of Mountain 
View, California. 

^5 Referring to FIGURE 1, each image layer 10 typically comprises the actual image information 12, compositing 

parameters 14, and, optionally, a mask 16. As is known in the art, the image 12 generally comprises a set of pixels in 
an mxn array. The compositing parameters 14 generally include such information as a transfer mode (e.g,,, color 
model), global opacity information, color-based restrictions on which areas are subject to blending, and. other informa- 
tion known in the art. 

20 A number of image layers, 1 . n, are merged to form a composite image. In the merge process, each pixel generally 

is independent of the other pixels of the image 12 within each image layer 10, so an implementing system can equally 
well calculate pixel data for the entire final image, or just a piece of the final image. FIGURE 2 is a block diagram 
showing diagrammatically that a set of image layers 20 are subjected to a compositing operation 22 to generate a 
merged image 24, in known fashion. The process of merger may also be represented by the following pseudocode: 

2S 

PROCEDURE CalculateMergedData (layers: ARRAY OF 
ImageLayer; result ; Image); 
VAR i: INTEGER; 
BEGIN 

30 InitializeMergedData (result) ; 

FOR i = 1 TO LEN (layers) DO 

Compo s i t e I mage I n t o I mage ( 
layers [i] .image, 

layers [i] . cotnpositiingParameters , 
35 layers [i] .mask, result) 

END FOR 
END CalculateMergedData; 



In the pseudocode above, the final image, "result", can be initialized either to a solid color or to a value representing 
40 transparency. 

In existing image manipulation programs, there is a large class of useful image effects based upon functions that 
are applied to each pixel of an image independently. These functions include operations like inverting the color at a 
pixel, increasing the brightness of all pixels, afc. Such functions typically are known as "adjustments", and are selectable 
via a user interface (e.g., menu or Icons) of an implementing program, such as Adobe Photoshop 3.0 referenced above. 
45 A key property that sets these functions apart from filters (such as blurring operations), is that each pixel in the result 
only depends on the spatially corresponding pixel in the source image. Adjustment operations generally involve ap- 
plying a particular function to some pixels based on some global parameters. In addltton, frequently a mask is specified 
to constrain the effect. An* adjustment operation is typically invoked by a call to a function such as the following, which 
specifies an adjustment function and parameters and the mask constraining that function: 

so 

App lyAd j ustmentTcI mage t 
adjustmentFunction, 
adjustmentParameters , 
55 mask, 

image) 

When applying an adjustment function, the mask is frequently accounted for by using separate images for the 
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initial and adjusted data, and then using the mask to compute a weighted average of the two images. This may require 
the use of a temporary buffer to hold a copy of the initial image, or to hold the adjusted data before blending it into the 
original data. This could, for example, result in a code path like the following: 

5 

ApplyAdjustmentNoMaskFromimageTo Image ( 

ad j us traentFunct ion , 

adj uscment Parameters , 
image , 

10 cmplmage) ; 

Blendlmages (tmplmage, image, mask, image) ; 

FIGURE 3A is a block diagrarh showing an image 30 to which is applied one or more adjustments 32 to generate 
an adjusted image 34, in accordance with the prior art. Thus, for example, an image X may be color corrected by an 
'5 adjustment to generate image X. 

FIGURE 3B is a block diagram showing diagrammaticaHy the application of a simple adjustment to a merged 
image, in accordance with the prior art. The adjustment data may be stored, for convenience, as a •pseudo^ayer" 36, 
so that the data is part of the array of actual image layers 1 ... n. The image layers 20 are merged by the application 
of a compositing operation 22 into a merged image 24. The adjustment function 36 is then applied 38 to the merged 
20 image 24 to generate an adjusted merged image 24*. Accordingly, adjustment operations can be readily incorporated 
into a conventional layer merging process in accordance with the following pseudocode; 
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TYPE 

Layers OBJECT 

mask: Image 

END; 

ImageLayer* OBJECT EXTENDING Layer 
image: Image; 

contpositingParameters : Com-^ 

posit ingParameCers 

END ; 

SimpleAdjustmentLayer- 

OBJECT EXTENDING Layer 

adjustment Function: Ad- 
j us tmentFunct ion ; 
adjustmentParameters : Ad- 
justment Parameters 

END; 

PROCEDURE CalGulateMergedData (layers; ARRAY OF Layer; 
result : Image) ; 
VAR i: INTEGER; 

BEGIN , ^ , 

InitializeMergedData (result) ; 
FOR i = 1 TO LBN (layers) DO 

IF layers [i] IS ImageLayer DO 

WITH layers [i] AS ImageLayer DO 
Compositelmagelntolmage ( 
layers [i] -image, 
layers [i] . compositingParamete- 
rs, 

layers [i] .mask, 
result ) 

END WITH 

ELSIF layers [i] IS SimpleAdj us tment Layer DO 
WITH layers [i] AS SimpleAdj ustmentLayer 
DO 

ApplyAdjustmentToImage ( 

layers [i] , adjustment Function, 
layers [i] .adjustmentParameters 

layers [i] .mask, 
result ) 

END WITH 

ELSE 

(* Unexpected layer type t *) 
END IF 
END FOR 
END calculateMergedData; 



(in the above pseudocode, types declared as OBJECTS are polymorphic records which ^^^9«"«'^' ^ f 
via DOinters When an object is declared as EXTENDING another OBJECT type, the object inherits all o the fields of 
re erendedXct type Thus, h the example above. OBJECTS "ImageLayers" and .SlmpleAdjustmentUyers' have 
■mask- ^eids which they inherit from OBJECT -Layer". The IS expression tests whether a particular object ,s really a 
specLdT^enstn of rts declared type. The WITH statement casts an object to an extension, allowing the program to 
access fields only defined in that extension - the IS test indicates that it is safe perform this cast) 

?n the above pseudocode, for each conventional image layer 1 ... n (having type -1 mage Layer" V conventional 
compos^^g 2 performed to generate the merged image 24. When the simple adjustment layer '^(f^^-^^^^ 
-srpTeAdjustmentLayer-) is detected, the adjustment function is applied 38 to the entire merged image 24 stored in 
variable "result" to generate the adjusted merged image 24' 
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A drawback of the prior products implementing simple adjustment layers is that they do not allow the system to 
edit the data underneath an adjustment "layer" while viewing the adjustment as applied to the final result. That is, the 
adjustment functbn does not truly have characteristics of an image layer. Further, the range of effects presented Is 
generally quite limited. 

Accordingly, it would be useful to have greater flexibility in applying adjustments to composited images. The present 
invention provides the system and method for allowing such flexibility. 

SUMMARY OF THE INVENTION 



'0 The present invention provides an advanced adjustment layer that may be applied during the compositing process 

to a set of image layers 1 ... n, or to any subordinate subset of such image layers. One or more adjustment layers are 
applied to an intermediate merged image, generated by compositfng previous image layers, and the adjusted result is 
stored as a temporary image. The temporary image is then in turn composited with the intermediate merged image. 
Any remaining image layers are then composited in with the intermediate merged image to generate a final merged 

^5 image. 

The invention allows a user to apply a vast array of effects without requiring significant new knowledge on the part 
ot the user. For example, if there are "A" adjustments and "T" transfer modes, the present invention allows AxT effects 
which leverage existing knowledge of the user of only A+T functions. 

The details of the preferred embodiment of the present invention are set forth In the accompanying drawings and 
20 ihe description below. Once the details of the invention are known, numerous additional innovations and changes will 
become obvious to one skilled in the art. 



BRIEF DESCRIPTION OF THE DRAWINGS 



25 FIGURE 1 is a diagram of a prior art image layer, showing the actual image information, compositing parameters, 

and a mask 

FIGURE 2 IS a block diagram showing diagrammatically that a set of image layers are subjected to a compositing 
operatton to generate a merged image, in accordance with the prior art. 

FIGURE 3A IS a block diagram showing an image to which is applied one or more adjustments to generate an 
30 adjusted image, in accordance with the prior art. 

FIGURE 3B is a block diagram showing diagrammatically the application of a simple adjustment to a merged 
innage. in accordance with the prior art. 

FIGURE 4 is a diagram of an adjustment layer comprising an adjustment function and adjustment parameters, in 
accordance with the present inventran. 
35 FIGURE 5 is a block diagram showing application of the present invention to a set of image layers 1 ... a of which 

at least one layer A is an adjustment layer in accordance with the present invention. 

Like reference numbers and designations in the various drawings indicate like elements. 

. DETAILED DESCRIPTION OF THE INVENTION 

40 

Throughout this description, the preferred embodiment and examples shown should be considered as exemplars, 
rather than as limitations on the present invention. 

In the present invention, instead of thinking of adjustments as operations that change pixels in a result image, 
adjustments can be thought of as producing new image data amenable to all of the compositing options available for 

45 manipulating image layers. Thus, the present invention uses an advanced image layer that permits great flexibility in 
controlling the adjustment and compositing process. 

FIGURES 4 and 5 are block diagrams depicting the architecture and use of the present invention. FIGURE 4 shows 
that an adjustment layer 40 in accordance with the present invention comprises an adjustment function 44 (for example, 
a color correction function) and optional adjustment parameters 46 (for example, parameters specifying the amount 

so of the color correction). Each adjustment layer 40 would also include all inherited structures inherent in being a layer 
{i.e., mask and compositing parameters). In some instances, or as a matter of design choice, adjustment layers 40 
may comprise only an adjustment function 44, with no adjustment parameters (an example would be simple color 
inversion for a specific color model). However, it is generally preferable to separate adjustment parameters from ad- 
justment functions so that the parameters are readily re-editable. 

55 An adjustment layer 40 is created in conventional fashion by allowing a user to select (e.g., by selecting from a 

menu or selecting an icon) an adjustment function {e.g., a color correction) and associated parameters. In the preferred 
embodiment, compositing parameters and adjustment parameters are re-editable after the creation of an adjustment 
layer Thus, a user can change the effect of an adjustment at any time. 



5 



BNSDOCID: <EP 0B271 12A2_I,> 



EP 0 827 112 A2 



in FIGURE 5 a set of image layers 52 comprises ^ ... n layers, of which at least one layer A is an adjustment layer 
The hierarchical position of the adjustment layer A within the set of image layers 1 ... n determines which image layers 
are operated on by the adjustment layer A In operation, image layers 1 .. . (M) are composited 53 into an intermediate 
merged image 54. The adjustments 55 defined by adjustment layer A are then applied to the intemnediate merged 
image 54 and the results are stored in a temporaiy buffer 56, The compositing operation 57 then continues, compositing 
the contents of temporary buffer 56 with the remaining image layers (A+l) ... ninto a final merged adjusted image 58. 
Note that the process is iterative if more than one adjustment layer A exists, and that image representation of the final 
merged adjusted image 58 may differ from the representation used for the image layers 52. 

The operations depicted in FIGURE 5 may also be described by the following pseudocode: 



10 



TYPE 

Lave r OBJECT ^ 

^ compositingParameters : Com- 

^5 posit ingParamecers ; 

mask: Image 

END; 

ImageLayer* OBJECT EXTENDING Layer 
image: Image; 

2ND ; 

20 

AdiustimentLayer* 

OBJECT EXTENDING Layer 

adj ustmentFunction : Ad- 
justment Function ; 
adjustmentParameters: Ad- 
25 j ustmentParameters 

END; 

30 Of thft procedure below to indicate 

that it may be allocated once at 
program initialization if it is 
deemed too expensive to allocate a 
temporary image every time the 
merge code is entered.*) 
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PROCEDURE CalculateMergedData (layers: ARRAY OF Layer; 

result : Image) ; 

VAR i: INTEGER; 

srclmage: Image; (♦ srclmage is the image that 

will be composited into the 
result. Note that Image is as- 
sumed to be a reference - 
i.e., a pointer type - and 
hence assigning to a variable 
of type Image is cheap, ♦) 

BEGIN 

InitializeMergedData (result) ; 
FOR i = 1 TO LEN{ layers) Do 

IF layers [i] IS ImageLayer DO 

WITH layers [i] AS ImageLayer DO 
srclmage = layers [ij . image 
END WITH 

ELSIF layers [i] is AdjustmentLayer DO 

WITH layers [i] AS AdjustmentLayer DO 
ApplyAd- 

j us tmentNoMaskFromlmageToImage ( 

layers (il . adjustmentFunction, 
layers [iJ . adjustment Parameters 

result, 
temp Image ) ; 
srclmage = templmage 
END WITH 

ELSE 

(* Unexpected layer type 2 *) 
END IF 

Compos it e Image Intolmage ( 
srclmage, 

layers [i] -compositingParameters , 
layers Ci] .mask, 
result ) 

END FOR 
END CalculateMergedData; 

In accordance with the above pseudocode, if an image layer is detected as being of type "ImageLayer", then it is 
simply composited in conventional fashion into "result*. However, if an image layer is of type "AdjustmentLayer", then 
the adjustment function defined within that layer is applied to the intermediate merged image 54 as it exists to that 
point in time. Of course, more than one adjustment layer may exist within the set of image layers 52, and the actual 
adjustment layer information may be stored in other forms and other locations. The results of the application of the 
adjustment layer are stored in a temporary buffer (variable templmage" in the pseudocode above), and then compos- 
ited into the intermediate merged image (variable "result"). The process continues until all image layers (including 
adjusted intermediate images) are composited. 

The invention allows a user to apply a vast array of effects without requiring significant new knowledge on the part 
of the user For example, if there are "A" adjustments and "T' transfer modes, the present invention allows AxT effects 
which leverage existing knowledge of the user of only A+T functions. 

As an example of using the present invention, an implementing system can create an effect based on inverting 
the color information in an image and then blending that result in such a way that the luminosity of the original color is 
preserved. As another example, a solarization-like effect can be accomplished by Inverting the color in an image and 
multiplying the color with itself. By using color restrictions on compositing, an implementing system can limit an adjust- 
ment effect to a dynamically calculated mask that can depend on either the p re-adjustment data, the post-adjustment 
data, or both. An implementing system can also gain code compactness by being able to mix and match adjustment 
logic with compositing logic. 
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AS Should be apparent, the invention allows for hierarchical compositing in which sub-ranges in stack of irnage 
lave^ 52 are composited together and the results are then composited in a next level of the hierarchy. When used 
wJh acjus^ent I^yTs^^^^ ability allows the system to restrict the effect of an adjustment layer so that .t does not 
affprt the entire imaae without having to create a special mask to do so. 

Afanom r aspSt of he present invention, the compositing parameters may be used to mask the compositing^ 
effect based on pixel-by-pixel properties of either the incoming image or the image to be merged (e.g the adjusted 
fncor^ing image ' pllmpl^ be based on ranges of color values. As an ^/^^ 1^^^^^^^^^^ 

Tele^ pixels with at least a certain luminance in the image for merging, or protect pixels with at least a certain level o 
colS in me iSoming image. Other masking functions can be used, like restricting composrting to colors inside or 
ourjde rprintabTe imut.Vhus, one can create an adjustment layer that applies a desaturation adjustment to any 

''''''i::^:;:;^.:™:^^! uses adjustment funct^ns that are applied on a one to one ^-is 
the conceot of the invention can be applied to multi-pixel adjustment functions, such as blurring, where each pixehs 
adU°S bled u^^^^^^^^^ Of other pixels, such as adjacent pixels. However, multi-pixel adjustment functions do 

fm so^^^^^^^^^^^ in that in implementing more generalized tigering operations as 1^^^^^^^^^^^ 
culate subareas within a merge image since the filter may require input areas that are larger than the area to be 
ca ifJaSd and th^t a series S filtering layers will produce a cascade effect ^--^^-^^^'^^^'^^^^ 
each other. When building an entire merged image, these issues are not a problem since all of the data is available, 
but they could impose processing costs when calculating small areas. 

Implementation 

The inventK^n may be implemented in hardware or software, or a combination of ^^^^^^^^^^ .J^^;^^^^^ 
inventKx, is implemented in computer programs executing on programmable computers each comp'^'"9 J one 
processor a data storage system (Including volatile and non-volatile memory and/or storage elennents). a» '^^s' . 
Tn^t d^ce Ind at least one output device. Program code is applied to input data to perform the functions described 
3.n and generate output inforJiation. The output information is applied to one or more output devices, in known 

'^'^^E'Tch program .s preferably implemented in a high level procedural or object oriented P^^Q^^'^'^'^Q '^^^'^ 
commtn!cat?wlth a computer system. However, the programs can be implemented in assembh. or machine language, 
if desired In any case, the language may be a compiled or interpreted language. ^^onetic diskette) 

Each such computer program is preferably stored on a storage media or device (e.g.. ROM or magnetic diskette, 
reac^b? b^a geS or special pur^se programmable computer, for configuring and operating the computer when 
t^^Saqe r^edl ^device is read by the computer to perfoon the procedures described herein. The invent^/e system 
rJay be cons^rL to be implemented as a computer-readable storage medium, <=f 'S^^f ^ -"^l^^VdZnne 
prSram where the storage medium so configured causes a computer to operate in a specrfic and predefined manner 

various nS» cations may be made wrthout departing from the spirit and scope of the invention. Accordingly rt is to be 
undtit^tSh"!nvemion is not to be limrted by the specific illustrated embodiment, but on^ by the scope of the 
appended claims. 

Claims 

A method for applying at least one adjustment during a process of compositing a set cf ordered image layers into 
a final merged image, comprising the steps of: 

(a) defining at least one adjustment layer within the ordered image layers, each adjustment layer comprising 

occurring before a current adjustment layer into a current intermediate 

Tc) Lp'f^S^o adjustment function of each current adjustment layer to the current intermediate merged image 

to generate a current adjusted temporary image; ^. . ^ 

(d) compositing each current adjusted temporary image into the current intermediate merged image^ 

(a) compositing any remaining image layers into the current intermediate merged image to create a final 

merged adjusted image. 
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A system for applying at least one adjustment during a process of compositing a set of ordered image layers into 
a final merged image, comprising: 

(a) means for defining at least one adjustment layer within the ordered image layers, each adjustment layer 
comprising at least an adjustment function: 

(b) means for compositing each image layer occurring before a current adjustment layer into a current inter- 
mediate-merged image; 

(c) .nieans for applying the adjustment function of each current adjustment layer to the current intermediate 
merged image to generate a current adjusted temporary image; 

(d) means for compositing each current adjusted temporary image into the current intermediate merged image; 

(e) means for compositing any remaining image layers into the current intermediate merged image to create 
a final merged adjusted image. - ■ - 

A computer program, residing on a computer-readable medium, for applying at least one adjustment during a 
process of compositing a set of ordered image layers into a final merged image, the computer program comprising 
instructions for causing a computer to: 

(a) define at least one adjustment layer within the ordered image layers, each adjustment layercomprising at 
least an adjustment function; 

(b) composite each image layer occurring before a current adjustment layer into a current intermediate merged 
image; 

(c) apply the adjustment function of each current adjustment layer to the current intermediate merged image 
to generate a current adjusted temporary image; 

(d) composite each current adjusted temporary image into the current intermediate merged image; 

(e) composite any remaining image layers into the current intermediate merged image to create a final merged 
adjusted image. 
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(54) Adjustment layers for composited image manipulation 



(57) A method and system for compositing graphical 
images, wherein an advanced adjustment layer may be 
applied during a compositing process to a set of image 
layers 1 ... n, ortoany subordinate subset of such image 
layers. One or more adjustment layers are applied to an 
intermediate merged image, generated by compositing 
previous image layers, and the adjusted result is stored 
as a temporary image. The temporary image is then in 



turn composited with the intemiediate merged image. 
Any remaining image layers are then composited in with 
the intermediate merged image to generate a final 
merged image. The invention allows a user to apply a 
vast array of effects without requiring significant new 
knowledge on the part of the user For example, if there 
are "A" adjustments and "T" transfer modes, the present 
invention allows AxT effects which leverage existing 
knowledge of the user of only A+T functions. 
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